Hybrid speaker and converter

ABSTRACT

An audio converter system is provided. The system comprises an audio input configured to receive a source audio, an audio output configured to couple to a hybrid speaker comprising at least two nondirectional speakers and a directional speaker, and a processor configured to generate an output audio for the hybrid speaker based on the source audio by: identifying a specific sound in the source audio, isolating the specific sound from the source audio, generating a directional speaker output for the directional speaker of the hybrid speaker based on the specific sound, and generating at least two channels of nondirectional speaker output for the at least two nondirectional speakers of the hybrid speaker.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/279,896, filed on Feb. 19, 2019, entitled “HYBRID SPEAKER ANDCONVERTER,” the entire contents and disclosure of which is hereby fullyincorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to audio processing, and morespecifically to generating audio signal for directional speakers.

2. Discussion of the Related Art

A loudspeaker (or speaker) is an electroacoustic transducer whichconverts an electrical audio signal into a corresponding sound. In aconventional dynamic cone speaker, for example, a coil of wire issuspended in a circular gap between the poles of a permanent magnet.When an alternating current electrical audio signal is applied, the coilis forced to move rapidly back and forth due to Faraday's law ofinduction, which causes a diaphragm attached to the coil to move backand forth, pushing on the air to disperse sound waves around thediaphragm.

SUMMARY OF THE INVENTION

One embodiment provides a system comprising an audio input configured toreceive a source audio, an audio output configured to couple to a hybridspeaker comprising at least two nondirectional speakers and adirectional speaker, and a processor configured to generate an outputaudio for the hybrid speaker based on the source audio by identifying aspecific sound in the source audio, isolating the specific sound fromthe source audio, generating a directional speaker output for thedirectional speaker of the hybrid speaker based on the specific sound,and generating at least two channels of nondirectional speaker outputfor the at least two nondirectional speakers of the hybrid speaker.

Another embodiment provides a method comprising identifying, with aprocessor, a specific sound in a source audio, isolating the specificsound from the source audio, generating a directional speaker output fora directional speaker of a hybrid speaker based on the specific sound,and generating at least two channels of nondirectional speaker outputfor at least two nondirectional speakers of the hybrid speaker.

Another embodiment provides an audio speaker apparatus comprising aspeaker housing, a left nondirectional speaker and a rightnondirectional speaker enclosed in the speaker housing, and adirectional speaker positioned between the left nondirectional speakerand the right nondirectional speaker and enclosed in the speakerhousing.

A better understanding of the features and advantages of variousembodiments of the present invention will be obtained by reference tothe following detailed description and accompanying drawings which setforth an illustrative embodiment in which principles of embodiments ofthe invention are utilized.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of embodiments ofthe present invention will be more apparent from the following moreparticular description thereof, presented in conjunction with thefollowing drawings wherein:

FIG. 1 is a block diagram illustrating a system for driving a hybridspeaker in accordance with some embodiments of the present invention;

FIG. 2 is a block diagram illustrating a system for converting audio fora hybrid speaker in accordance with some embodiments of the presentinvention;

FIG. 3 is a flow diagram showing a method for converting source audiofor a hybrid speaker in accordance with some embodiments of the presentinvention; and

FIG. 4A and FIG. 4B are illustrations of directional speakers with aplurality of panels in accordance with some embodiments of the presentinvention.

DETAILED DESCRIPTION

Directional speakers generally refer to devices that create a field ofsound that spread less than traditional speakers. In some embodiments, adirectional speaker may comprise a parametric array speaker that createssound from ultrasound. Directional speaker devices achieve highdirectivity by modulating audible sound onto high-frequency ultrasound.The higher frequency sound waves have a shorter wavelength and thus donot spread out as rapidly, thus resulting in directivity that is higherthan conventional nondirectional loudspeakers.

In some embodiments described herein, a directional speaker is combinedwith one or more conventional nondirectional speakers to simulateimmersive sound and/or to selectively project select sound to specificuser(s) in a room. In some embodiments, the directional andnondirectional speakers may be enclosed in the same housing to provide ahybrid soundbar. In some embodiments, a converter is provided to convertconventional audio file and/or stream to signals for driving a hybridspeaker. The converter may be configured to identify and isolatespecific sound to be reproduced by the directional speaker and generateoutputs for the directional and nondirectional speakers in the speakerassembly.

Referring first to FIG. 1 , there is shown a system for driving a hybridspeaker. The system comprises an audio source 110, an audio converter120, and a hybrid speaker assembly 100 producing sound for a user 130.

The audio source 110 generally refers to a device that outputs a soundfor playback at one or more speakers. In some embodiments, the audiosource 110 may comprise a playback device such as a television, a cablebox, a media player, a hard drive, a media storage device, a homeentertainment system, a computer, a game console, a mobile device, andthe like. The audio source 110 may output mono sound, stereo sound,and/or surround sound. In some embodiments, the audio source 110 mayoutput one or more nondirectional audio channels. In some embodiments,the audio source 110 may output any number of audio channels, such astwo channels (e.g. left/right stereo sound), three channels (e.g.center, left, and right), six channels (e.g. 5.1 surround), eightchannels (e.g. 7.1 surround), etc.

The audio converter 120 comprises a device configured to convert audiosignal from the audio source 110 to signals configured to drivedirectional and nondirectional speakers of the hybrid speaker assembly100. In some embodiments, the audio converter 120 is configured to takeone or more conventional audio channels and generate at least one outputchannel for a directional speaker 103. In some embodiments, the audioconverter 120 comprises a processor configured to identify a specificsound in the source audio, isolate the specific sound from the sourceaudio, generate a directional speaker output for the directional speakerof the hybrid speaker based on the specific sound, and generate at leasttwo channels of nondirectional speaker output for the at least twonondirectional speakers of a hybrid speaker. In some embodiments, theaudio converter 120 may further comprise a sound database foridentifying a specific sound in the source audio. In some embodiments,the sound database may comprise a cloud database that is shared among aplurality of audio converter 120 and accessible via a network adapter onthe audio converter 120 configured to communicate over a network such asa home network and/or the Internet. In some embodiments, the audioconverter 120 may be configured to run a machine learning algorithm foridentifying the specific sound based on a growing and evolving learningset of sounds stored in the sound database. Further details of the audioconverter 120 are provided with reference to FIGS. 2 and 3 herein.

The hybrid speaker assembly 100 generally refers to a speaker devicecomprising at least one directional and one or more nondirectionalspeakers. In the embodiment shown in FIG. 1 , the hybrid speakercomprises a left nondirectional speaker and a right nondirectionalspeaker enclosed in the speaker housing and a directional speakerpositioned between the left nondirectional speaker and the rightnondirectional speaker. In some embodiments, the hybrid speaker assembly100 may comprise the form factor of a soundbar. In some embodiments, ahybrid speaker assembly 100 may comprise two or more separately movablehousings.

The left speaker 101 and the right speaker 102 comprise nondirectionalspeakers. Nondirectional speaker generally refers to conventionalspeakers which have a large spread and small directionality. Forexample, the sound produced by a nondirectional speaker may generally besimilarly audible to listeners positioned in any direction relative tothe speaker. In some embodiments, the left speaker 101 and the rightspeaker 102 may comprise a dynamic loudspeaker comprising a diaphragmconnected to a frame via a flexible suspension that constrains a voiceoil. In some embodiments, the left speaker 101 and the right speaker 102may comprise speaker elements similar to those in a conventionalsoundbar. In some embodiments, the hybrid speaker assembly 100 maycomprise additional nondirectional speakers such as a center speaker, asubwoofer, and the like.

The directional speaker 103 generally refers to a speaker device with anarrow spread and high directionality. Generally, sound projected by adirectional speaker is only audible within a specific direction relativeto the speaker. In some embodiments, the directional speaker 103 may beconfigured to carry audible sound wave on inaudible high-frequencywaveform (e.g. ultrasound). In some embodiments, the directional speaker103 may comprise a parametric array speaker that creates sound fromultrasound.

In some embodiments, the directional speaker 103 may comprise aplurality of fixed panels. For example, the directional speaker 103 maycomprise two or more fixed parametric array panels that are angledrelative to each other. An example of a two-panel directional speakerconfiguration is shown in FIG. 4A and an example of a three-paneldirectional speaker configuration is shown in FIG. 4B. The panels of thedirectional speaker 103 may be separately controlled to affect theoverall projection direction of the directional speaker 103. Forexample, the outputs of each panel may be modulated to cause theprojection direction of the directional speaker 103 to sweep acrossmultiple angles in a projection area to selectively direct sound atdifferent intended directions. In some embodiments, the sweep may beperformed at a speed that is recognizable by human ears. In someembodiments, the speed sweep may be between 0.01 Hz to 10 Hz or faster.

The projection angles controlled by modulating the output of thedifferent panels may also be used to separately direct sound towards twoor more listeners sitting apart and/or create simulated surround soundeffect by projecting different sound towards the left and right ears ofa listener. In some embodiments, each panel of the directional speaker103 may be driven by a separate audio stream outputted from the audioconverter 120. In some embodiments, the hybrid speaker assembly 100and/or the audio converter 120 may be configured to convert audio andprojection direction data into audio output for each panel of thedirectional speaker 103. For example, sound modulation may process asound signal dynamically over time. using phase shifts and/or amplitudecontrol to provide the sound effect by sweeping the signal fromleft/right to right/left.

In some embodiments, the directional speaker 103 may comprise amechanical rotor configured to physically rotate the directional speaker103 to control the projection direction of the speaker. For example, therotor may turn a parametric array towards the user 130 to direct thesound to the user. In some embodiments, the rotor may sweep theprojection direction of the directional speaker 103 across a projectionarea, and the directional speaker 103 may only produce sound at intendedangles during the rotation such that a single directional speaker may beperceived to project sound at multiple discrete or continuousdirections. For example, the directional speaker 103 may directdifferent sounds to different listeners sitting apart in a room bytiming the output audio to the rotation. In another example, thedirectional speaker 103 may direct different sounds toward the left andright ears of a user to simulate surround or immersive sound. In someembodiments, the directional speaker 103 is driven by a signal thatcomprises audio and projection direction data associated with one ormore temporal portions of audio, and the rotor may be controlled by theprojection direction data associated with the audio signal.

While only one directional speaker 103 is shown in the hybrid speakerassembly 100, in some embodiments, the hybrid speaker assembly 100 maycomprise a plurality of directional speakers 103 driven by differentinput. For example, the hybrid speaker may comprise two or moreparametric arrays mounted on separate rotors and/or a plurality of setsof parametric array panels. While the audio converter 120 is shownseparately from the hybrid speaker assembly 100, in some embodiments,the audio converter 120 may be integrated into the housing of the hybridspeaker assembly 100 as one physical device. In some embodiments, theaudio converter 120 may be local to the hybrid speaker assembly 100and/or comprise a cloud-based and/or server-based converter. In someembodiments, the audio converter 120 may be integrated with an audiosource 110, such as a home entertainment console, a game console, acomputer system, and the like. The audio source 110 may selectivelyengage the audio converter 120 based on the audio format (e.g.conventional audio or hybrid speaker ready audio format) and/or thespeaker device coupled to the audio source 110 (e.g. conventionalspeaker or hybrid speaker).

With the system shown in FIG. 1 , a conventional audio source 110 may beconverted, via the audio converter 120, to signal for driving a hybridspeaker assembly 100 comprising a directional speaker 103. With a singleuser 130, the directional speaker 103 may direct different sound to theleft and right sides of the user that is mixed with the output from theconventional speakers to simulate surround and/or immersive soundeffect. In some embodiments, the directional speaker 103 may beconfigured to project sounds only audible to specific listener(s) in aprojection area while the conventional speakers generate sounds audibleto all listeners. For example, in a multiplayer game, different specificsounds (e.g. voice instructions, status alerts, warnings) may beprojected to different players while ambient sound or background musicmay be played through the nondirectional speakers.

In some embodiments, the hybrid speaker assembly 100 may be used withoutthe audio converter 120. In some embodiments, an audio source 110 mayoutput audio configured for a hybrid speaker assembly 100 comprisingdirectional and nondirectional speaker signals and the audio converter120 may be omitted. A hybrid speaker compatible game console, forexample, may output select sound files (e.g. voice instructions, statusalerts, warnings) via a directional speaker channel and the remainingsound files (e.g. background music) via one or more nondirectionalspeaker channel in its audio output. In another example, prerecordedaudio (e.g. movie, music, etc.) may be recorded with at least onedirectional audio channel that can be directly used by a hybrid speakerwithout conversion.

Referring next to FIG. 2 , there is shown a system for converting audiosource for use by a hybrid speaker. The audio converter 200 is generallyconfigured to take signals from an audio source 210 and output signalsto the nondirectional speaker 221 and the directional speaker 222 of ahybrid speaker assembly 220. In some embodiments, the audio source 210,the audio converter 200, and the hybrid speaker assembly 220 maycomprise the audio source 110, the audio converter 120, and the hybridspeaker assembly 100 described with reference to FIG. 1 herein.

The audio converter 200 comprises an audio input 201, an identificationmodule 202, a sound database 203, a sound filter 204, a nondirectionalspeaker output 205, a projection direction module 206, and a directionalspeaker output 208.

The audio input 201 is generally configured to receive analog or digitalaudio signal from an audio source 210. The audio input 201 may comprisewired and/or wireless connectors such as an audio connector, anaudio/video connector, an HDMI connector, a USB connector, a Bluetoothtransceiver, a wireless HDMI transceiver, and the like. In someembodiments, the audio input 201 may further comprise other componentsfor filtering, converting, and/or boosting the input signal for the restof the circuitry for processing.

The identification module 202 is generally configured to identify aspecific sound in the input audio for playback through the directionalspeaker. The identification module 202 may be implemented on aprocessor, a control circuit, an application specific integrated circuit(ASIC), and/or one or more digital or analog audio filter components. Insome embodiments, the specific sound is identified by comparing thesource audio with sound clips and/or sound profiles in a sound database203. For example, the sound database 203 may comprise sound clip and/orsound profiles (e.g. frequency spectrum characteristics, frequencydistribution, temporal frequency or amplitude changes, etc.) associatedwith a plurality of types of sounds suitable for playback via adirectional speaker. Examples of such sound may include car sounds, birdsounds, ballistic projectile sounds, human speech, alerts, warningsounds, and the like. In some embodiments, the identification module 202may comprise audio event detection algorithms which utilizesconvolutional neural network (CNN), recurrent neural network (RNN),and/or generative adversarial network (GAN) that identify specificsounds through machine learning using sounds in the sound database 203as a learning set. In some embodiments, listener response may be used asfeedback to the machine learning algorithm. In some embodiments, othertypes of feedback used for machine learning may comprise camera input,face detection/tracking, and/or object detection/tracking. In someembodiments, user and environmental feedback may be used in directionand volume control. In some embodiments, the specific sound may beidentified based on frequency, amplitude, and/or duration. For example,the sound identification module 202 may be configured to detect soundswith high frequency and short duration as specific sound for playbackvia a directional speaker. In some embodiments, the specific sound maybe identified based on at least one of sound frequency and amplitudedifference between two or more channels of the source audio. Forexample, if a sound within the sound stream is louder (e.g. havinginteraural intensity difference (IID) or time delayed (e.g. havinginteraural time difference (ITD)) between the left or right speakerchannels, the identification module may identify the sound associatedwith significant IID or ITD for playback at the directional speaker. Insome embodiments, a sound may be initially identified based on IIDand/or ITD and added to the sound database 203. The identificationmodule 202 may then use the sound samples in the sound database 203 toidentify similar sounds in the source audio that may not havesignificant IID and/or ITD.

In some embodiments, the sound database 203 may comprise a local and/ora network-based database storing audio clips and/or sound profiles. Insome embodiments, the sound database 203 may be maintained by a centralservice shared among a plurality of audio converters 200 associated withdifferent users and different spaces. In some embodiments, the sounddatabase 203 may be updated and pruned by machine learning algorithmsuch as a generative adversarial network (GAN) using audio and feedbackreceived at a plurality of audio converters. In some embodiments, theaudio converter 200 may comprise a network interface (e.g. networkadapter, wireless transceiver, etc.) for communicating with anetwork-based sound database 203. In some embodiments, at least aportion of the function of the identification module 202 and the soundfilter 204 may be carried out by a remote server via the networkinterface.

The sound filter 204 is configured to isolate the specific soundidentified by the identification module 202 from the other sounds in theaudio source 210. In some embodiments, the sound filter 204 may usetemporal filtering, amplitude filtering, and/or frequency filtering suchas high, low, or band filtering to isolate the sound. In someembodiments, the sound filter 204 may use the difference betweendifferent channels of the audio source 210 to isolate the specificsound. For example, if a high degree of ITD or IID is detected betweenchannels of the input audio stream, one channel may be used to isolatethe specific sound in another channel by subtraction or masking. In someembodiments, the filtering may only approximate isolation of thespecific sound; as sound from the directional speaker's would be mixedwith the sound from the nondirectional speaker during playback,incomplete isolation may not significantly affect user experience. Insome embodiments, a sound from the sound database 203 that matched thespecific sound may be used to filter the sound and/or recreate thespecific sound. The sound filter 204 then provides the specific soundisolated from the input audio to the projection direction module 206 andthe directional speaker output 208. The sound filter 204 may furthersubtract the specific sound from one or more channels of the sourceaudio and output the modified nondirectional audio signal to thenondirectional speaker(s) 221 of the hybrid speaker. In someembodiments, the source audio may be provided to the nondirectionalspeakers 221 without modification.

The nondirectional speaker output 205 is generally configured to coupleto the hybrid speaker assembly 220 and provide audio signal for playbackat the nondirectional speaker(s) 221. In some embodiments, thenondirectional speaker output 205 may comprise wired and/or wirelessconnectors such as an audio connector, an audio/video connector, an HDMIconnector, a USB connector, a Bluetooth transceiver, a wireless HDMItransceiver, and the like. In some embodiments, the nondirectionalspeaker output 205 may further comprise other components for providingthe signal to the speaker such as an amplifier, a digital to analogconverter and the like. In some embodiments, the nondirectional speakeroutput 205 may output a plurality of channels each corresponding to anondirectional speaker (e.g. right speaker, left speaker) of the hybridspeaker assembly 220 via a single wire, multiple wires, or a wirelessconnection.

The projection direction module 206 is configured to determine aprojection direction associated with the output to the directionalspeaker 222. In some embodiments, the projection direction module 206determines a projection direction based on one or more of a userlocation and a time delay or/or amplitude difference between two or morechannels of the source audio. In some embodiments, the projectiondirection may be determined based on the user's location detected by auser sensor 224 on the hybrid speaker assembly 220 and/or on the audioconverter 200 such that the directional speaker projects the specificsound towards one or more users. In some embodiments, user locationand/or projection area may be determined through a speaker calibrationprocess. In some embodiments, the projection direction module 206 may bedetermined based on the amplitude (e.g. loudness) difference of thespecific sound in the two or more channels (IID) of the source audioand/or the time delay difference of the specific sound between the twoor more channels of the source audio (ITD). For example, if the specificsound is significantly louder on the left channel, the projectiondirection of the directional speaker 222 may be angled to the left. Inanother example, if the amplitude and timing of the specific sound aresubstantially the same in both channels, the projection direction of thedirectional speaker 222 may be directed towards the center. In someembodiments, the projection direction module 206 may determine a 2Dsoundscape to be outputted by the directional speaker 222 and determinesound to be outputted at each angle as the directional speaker 222sweeps through the projection area of the 2D soundscape. In someembodiments, the directional speaker 222 may be configured sweep throughthe soundscape space in a fixed pattern, and the projection directionmodule 206 is configured to time the audio signal output to the sweepposition of the directional speaker 222 to output the 2D soundscape. Insome embodiments, an optical sensor may be used to detect user locationand user detection/tracking information may be used to determineprojection direction. In some embodiments, user location may bedetermined through a calibration process and/or be manually adjusted bya user.

The directional speaker output 208 is generally configured to couple tothe hybrid speaker and provide a signal for playback at the directionalspeaker(s) 222 of the hybrid speaker assembly 220. In some embodiments,the directional speaker output 208 may comprise wired and/or wirelessconnectors such as an audio connector, an audio/video connector, an HDMIconnector, a USB connector, a Bluetooth transceiver, a wireless HDMItransceiver, and the like. In some embodiments, the directional speakerinput 201 may further comprise other components for providing the signalto the speaker such as an amplifier, a digital to analog converter, andthe like. In some embodiments, the output signal may comprise audiosignals synchronized to the sweep of the directional speaker 222. Insome embodiments, the output signal may comprise one or more projectionangles each associated with a temporal portion of an audio stream. Forexample, the audio output may specify that the first 2 nanoseconds ofaudio is to be projected at 35 degrees, and the next 3 nanoseconds ofaudio is to be projected at 50 degrees. In some embodiments, the outputsignal may comprise an audio stream and a directional speaker movementinstruction timed to the audio stream. For example, directional speakermovement instruction may specify the speed of the sweep, the range ofthe sweep, etc. In some embodiments, for a directional speaker 222 witha mechanical rotor, the audio converter may output a rotor controlsignal separately from the audio stream. In some embodiments, for adirectional speaker 222 with a plurality of fixed panels, the outputsignal may comprise separate audio channels to be outputted at each ofthe panels of the directional speaker 222 that is modulated to controlthe projection direction of the directional speaker 222. In someembodiments, the output signal may comprise a 2D soundscape for playbackby the directional speaker 222 and the hybrid speaker assembly 220 maycomprise a processor configured to control the projection direction andthe audio output of the directional speaker 222 to reproduce the 2Dsoundscape.

In some embodiments, the outputs from the nondirectional speaker output205 and directional speaker output 208 are timed such that the outputsfrom the nondirectional speaker 221 and the directional speaker 222 aresynchronized and mixed when the sounds for each speaker device reach thelistener. In some embodiments, machine learning such as GAN may beapplied to selectively mix and synchronize the output to thenondirectional speaker 221 and the directional speaker 222 using thesound database 203.

In some embodiments, one or more functions of the projection directionmodule 206 described herein may be carried out by the hybrid speakerassembly 220 instead. For example, in embodiments where the directionalspeaker 222 is configured to project the specific sound based solely onuser location, the directional speaker output 208 may only output thespecific sound and the hybrid speaker assembly 220 may determine theprojection direction based on the user location detected by the usersensor 224. In such case, the user sensor 224 may not provide userlocation information to with the audio converter 200 and the projectiondirection module 206 may be omitted on the audio converter 200.

The user sensor 224 is generally configured to detect the location ofone or more listeners near the directional speaker. In some embodiments,the user sensor 224 may comprise an optical sensor, a microphone, asonic sensor, an ultrasound sensor, a thermal sensor, and the like. Insome embodiments, the user location may be determined by detecting forsound reflection from a sound produced by the directional speaker. Forexample, the directional speaker 222 may sweep a projection area withdirectional sound and a user sensor 224 comprising a microphone maymeasure the timing of sound reflection at each angle to determine thepresence and/or location of user(s).

In some embodiments, the connection between the audio converter 200 andthe hybrid speaker assembly 220, while shown with multiple arrows, maycomprise a single wire or a wireless connection. In some embodiments,the audio converter 200 and the hybrid speaker assembly 220 may be anintegrated device enclosed in a single housing. In some embodiments, theaudio converter 200 may be integrated with an audio source 210 thatselectively engaged the audio converter based on audio format and/orspeaker type.

Referring next to FIG. 3 , there is illustrated an example of a methodfor converting audio for a hybrid speaker. In some embodiments, thesteps shown in FIG. 3 may be performed by one or more of an audioconverter, a hybrid speaker controller, a computer system, and a server.In some embodiments, one or more the steps shown in FIG. 3 may beperformed by the audio converter 120 and/or the hybrid speaker assembly100 described with reference to FIG. 1 and/or the audio converter 200and/or the hybrid speaker assembly 220 described with reference to FIG.2 herein. Generally, the steps in FIG. 3 may be performed by one or moreprocessor-based devices comprising a control circuit and a memorystoring a set of computer-executable instructions.

In step 301, the system identifies a specific sound in audio providedfrom the audio source 310. The specific sound generally refers to asound that is selected for playback through the directional speaker. Insome embodiments, the specific sound is identified by comparing thesource audio with sounds and/or sound profiles in a sound database 312.In some embodiments, the specific sound is identified through machinelearning algorithm 311 using the sound database 312 as a learning set.In some embodiments, the machine learning algorithm 311 may comprise agenerative adversarial network (GAN). In some embodiments, the specificsound is identified based on one or more of the frequency, the tone, andthe amplitude of the sound. In some embodiments, the sound database 312may comprise sound clip and/or sound profiles (e.g. frequency spectrumcharacteristics, frequency distribution, temporal frequency or amplitudechanges, etc.) associated with a plurality of types of sounds suitablefor playback via a directional speaker. Examples of such sounds mayinclude car sounds, bird sounds, ballistic projectile sounds, humanspeech, alerts, warning sounds, and the like. In some embodiments, asound may be initially identified based on frequency, duration,amplitude, and/or IID and/or ITD between two or more channels of thesource audio and added to the sound database 312. The sound may be usedby machine learning algorithm 311 to identify other similar sounds. Insome embodiments, the sound database 312 may be prioritized and/orpruned by a machine learning algorithm. In some embodiments, step 301may be performed by the identification module 202 described withreference to FIG. 2 herein.

In step 302, the system isolates the specific sound. Generally, in step302, the specific sound identified in step 301 is isolated from theother sounds in audio received from the audio source 210. In someembodiments, step 302 may be performed based on temporal filtering,amplitude filtering, and/or frequency filtering such as high, low, orband filtering. In some embodiments, the system may use differentchannels of the audio source 310 to isolate the specific sound. Forexample, if a large IID or ITD is present between channels of the inputaudio stream, one channel may be used to isolate the specific sound inanother channel by subtraction or masking. In some embodiments, thespecific sound may be isolated using the matching sound in the sounddatabase 312 by subtraction or masking. In some embodiments, onlyapproximate isolation of the specific sound is performed in step 302. Insome embodiments, the specific sound may be recreated using the soundsin the sound database. For example, the playback speed, the tone, and/orthe amplitude of a sound in the sound database may be modified toapproximate the specific sound identified in the audio source 310. Insome embodiments, step 302 may be performed by the sound filter 204described with reference to FIG. 2 herein.

In step 303, the system generates output for the directional speaker321. In some embodiments, the output comprises the specific soundisolated in step 302. In some embodiments, the directional speaker maybe configured to perform a periodic sweep of a projection area and thedirectional speaker audio output may be timed to the periodic sweep. Insome embodiments, the directional speaker output may further compriseprojection direction data. In some embodiments, the projection directiondata may be determined by the projection direction module 206 describedwith reference to FIG. 2 . In some embodiments, the projection directionmay be associated with user location. In some embodiments, theprojection direction data may comprise two or more discreet projectionzones associated with two or more users such that different sound may beprojected to different users in the same room using one directionalspeaker. In some embodiments, projection direction data may associateprojection angles with temporal portions of the directional speakeraudio. In some embodiments, the directional speaker output may comprisea 2D soundscape to be produced by the directional speaker. In someembodiments, the directional speaker output may comprise separate audiostreams for each panel of a fixed multipaneled directional speaker. Insome embodiments, the directional speaker output may comprise rotationinstructions for a rotor-controlled directional speaker.

In step 304, the system generates output for the nondirectional speaker322. In some embodiments, directional speaker output may comprise theaudio source 310 with the specific sound subtracted. In someembodiments, the directional speaker output may correspond to unalteredsource audio. In some embodiments, the nondirectional speaker output maycomprise two or more channels each corresponding to a nondirectionalspeaker device (e.g. left speaker, right speaker) in the hybrid speakerassembly.

In some embodiments, the steps shown in FIG. 3 may be repeatedcontinuously for an audio stream in near real time such that theconverted audio is synchronized with video playback associated with thesource audio. In some embodiments, the system may pre-buffer the sourceaudio to allow for video/audio playback synchronization. In someembodiments, the steps shown in FIG. 3 may be performed on a prerecordedaudio file. In some embodiments, instead of outputting to a speaker, thegenerated directional speaker output and nondirectional speaker outputmay be stored as an audio file for playback at a later time.

FIGS. 4A and 4 B are illustrations of examples of multi-paneldirectional speakers according to some embodiments. FIGS. 4A and 4Bcomprise top views of the panels of a directional speaker arranged toproject sound at multiple angles across a projection area 420 throughmodulation. In some embodiments, panels 410 a-e may each comprise aparametric array configured to project a narrow spread of sound throughultrasound.

In FIG. 4A, a two-panel configuration is shown. The panels 410 a and 410b are positioned at an angle relative to each other and between anondirectional left speaker 430 and right speaker 431. The angle betweenthe two panels may be between 170-90 degrees. The two panels 410 a and410 b are configured to change the projection direction of the soundwithin the projection area 420 through modulating the audio output ofthe two panels.

In FIG. 4B, a three-panel configuration is shown. The panels 410 c, 410d, and 410 e are positioned at an angle towards each other and between anondirectional left speaker 430 and right speaker 431. In someembodiments, the center panel 410 d is generally parallel to a linedrawn between nondirectional left 430 and right speakers 431 of aspeaker assembly. The angle between each of the side panels 410 c and410 e and the center panel 410 d may be between 170-70 degrees. Thethree panels 410 c-410 e are configured to change the projectiondirection of the sound within the projection area 420 through modulatingthe audio output of the three panels.

In some embodiments, a hybrid speaker may comprise directional speakersin other configurations. For example, a directional speaker may comprisefour or more parametric panels. In another example, a directionalspeaker may comprise one or more curved parametric panel. In yet anotherexample, the directional speaker may comprise a 2D matrix of panels(e.g. 2×2, 3×3, etc.) to project sound in two axes (e.g. left-right andtop-down). In some embodiments, the hybrid speaker may comprise adirectional speaker driven by a mechanical rotor configured to turnand/or tilt one or more parametric arrays. The distances, proportions,and relative positions of elements in FIGS. 4A and 4B are provided forreference only and are not necessarily drawn to scale.

In some embodiments, one or more of the embodiments, methods,approaches, and/or techniques described above may be implemented in oneor more computer programs or software applications executable by aprocessor-based apparatus or system. By way of example, such processorbased apparatuses or systems may comprise a computer, entertainmentsystem, game console, workstation, graphics workstation, server, client,portable device, pad-like device, etc. Such computer program(s) may beused for executing various steps and/or features of the above-describedmethods and/or techniques. That is, the computer program(s) may beadapted to cause or configure a processor-based apparatus or system toexecute and achieve the functions described above. For example, suchcomputer program(s) may be used for implementing any embodiment of theabove-described methods, steps, techniques, or features. As anotherexample, such computer program(s) may be used for implementing any typeof tool or similar utility that uses any one or more of theabove-described embodiments, methods, approaches, and/or techniques. Insome embodiments, program code macros, modules, loops, subroutines,calls, etc., within or without the computer program(s) may be used forexecuting various steps and/or features of the above-described methodsand/or techniques. In some embodiments, the computer program(s) may bestored or embodied on a computer readable storage or recording medium ormedia, such as any of the computer readable storage or recording mediumor media described herein.

Therefore, in some embodiments the present invention provides a computerprogram product comprising a medium for embodying a computer program forinput to a computer and a computer program embodied in the medium forcausing the computer to perform or execute steps comprising any one ormore of the steps involved in any one or more of the embodiments,methods, approaches, and/or techniques described herein. For example, insome embodiments the present invention provides one or morenon-transitory computer-readable storage mediums storing one or morecomputer programs adapted or configured to cause a processor-basedapparatus or system to execute steps comprising: rendering a computersimulated scene for display to a user, detecting an onset of a saccadethat causes saccadic masking in an eye movement of the user viewing thecomputer simulated scene, and reducing a computing resource used forrendering frames of the computer simulated scene during at least aportion of a duration of the saccade.

While the invention herein disclosed has been described by means ofspecific embodiments and applications thereof, numerous modificationsand variations could be made thereto by those skilled in the art withoutdeparting from the scope of the invention set forth in the claims.

What is claimed is:
 1. An audio converter system, comprising: an audioinput configured to receive a source audio; and a processor configuredto generate an output audio based on the source audio by performingsteps comprising: identifying a specific sound in the source audio,wherein the specific sound is identified through machine learning usingsounds in a sound database as a learning set, and wherein the sounddatabase is updated by machine learning; generating an output for adirectional speaker based on the specific sound; and projecting theoutput of the directional speaker so that the output is audible to onlyone or more specific users in a room and not audible to other users inthe room.
 2. The system of claim 1, wherein the machine learning isbased on a generative adversarial network (GAN).
 3. An audio convertersystem, comprising: an audio input configured to receive a source audio;and a processor configured to generate an output audio based on thesource audio by performing steps comprising: identifying a specificsound in the source audio; generating an output for a directionalspeaker based on the specific sound; and projecting the output of thedirectional speaker so that the output is audible to only one or morespecific users in a room and not audible to other users in the room;wherein the specific sound is identified based on sound frequency and/oramplitude difference between two or more channels of the source audio.4. The system of claim 1, wherein the processor is further configured toperform steps comprising: determining a projection direction for thedirectional speaker based on a user location, an amplitude differencebetween two or more channels of the source audio, and/or a time delaydifference between two or more channels of the source audio.
 5. Thesystem of claim 4, wherein the projection direction for the directionalspeaker is outputted to the directional speaker to control a mechanicalrotor configured to rotate the directional speaker.
 6. The system ofclaim 1, wherein the processor is further configured to perform stepscomprising: generating an output for at least one nondirectionalspeaker; and projecting the output of the at least one nondirectionalspeaker so that the output is audible to all users in the room.
 7. Thesystem of claim 6, wherein the output for at least one nondirectionalspeaker is generated based on subtracting the specific sound from thesource audio.
 8. The system of claim 6, wherein the generating an outputfor at least one nondirectional speaker comprises: generating at leasttwo channels of nondirectional speaker output for at least twonondirectional speakers.
 9. A method, comprising: identifying a specificsound in a source audio, wherein the specific sound is identifiedthrough machine learning using sounds in a sound database as a learningset, and wherein the sound database is updated by machine learning;generating an output for a directional speaker based on the specificsound; and projecting the output of the directional speaker so that theoutput is audible to only one or more specific users in a room and notaudible to other users in the room.
 10. The method of claim 9, whereinthe machine learning is based on a generative adversarial network (GAN).11. A method, comprising: identifying a specific sound in a sourceaudio; generating an output for a directional speaker based on thespecific sound; and projecting the output of the directional speaker sothat the output is audible to only one or more specific users in a roomand not audible to other users in the room; wherein the specific soundis identified based on sound frequency and/or amplitude differencebetween two or more channels of the source audio.
 12. The method ofclaim 9, further comprising: determining a projection direction for thedirectional speaker based on a user location, an amplitude differencebetween two or more channels of the source audio, and/or a time delaydifference between two or more channels of the source audio.
 13. Themethod of claim 12, wherein the projection direction for the directionalspeaker is outputted to the directional speaker to control a mechanicalrotor configured to rotate the directional speaker.
 14. The method ofclaim 9, further comprising: generating an output for at least onenondirectional speaker; and projecting the output of the at least onenondirectional speaker so that the output is audible to all users in theroom.
 15. The method of claim 14, wherein the output for at least onenondirectional speaker is generated based on subtracting the specificsound from the source audio.
 16. The method of claim 14, wherein thegenerating an output for at least one nondirectional speaker comprises:generating at least two channels of nondirectional speaker output for atleast two nondirectional speakers.
 17. A non-transitory computerreadable storage medium storing one or more computer programs configuredto cause a processor-based system to execute steps comprising:identifying a specific sound in a source audio, wherein the specificsound is identified through machine learning using sounds in a sounddatabase as a learning set, and wherein the sound database is updated bymachine learning; generating an output for a directional speaker basedon the specific sound; and projecting the output of the directionalspeaker so that the output is audible to only one or more specific usersin a room and not audible to other users in the room.
 18. Thenon-transitory computer readable storage medium of claim 17, wherein theone or more computer programs are further configured to cause theprocessor-based system to execute steps comprising: generating an outputfor at least one nondirectional speaker; and projecting the output ofthe at least one nondirectional speaker so that the output is audible toall users in the room.
 19. The non-transitory computer readable storagemedium of claim 18, wherein the output for at least one nondirectionalspeaker is generated based on subtracting the specific sound from thesource audio.
 20. The non-transitory computer readable storage medium ofclaim 18, wherein the generating an output for at least onenondirectional speaker comprises: generating at least two channels ofnondirectional speaker output for at least two nondirectional speakers.21. An audio system, comprising: a speaker housing; a directionalspeaker enclosed in the speaker housing; and an audio converterconfigured to identify a specific sound in a source audio, generate anoutput for the directional speaker based on the specific sound, andproject the output of the directional speaker so that the output isaudible to only one or more specific users in a room and not audible toother users in the room, wherein the specific sound is identifiedthrough machine learning using sounds in a sound database as a learningset, and wherein the sound database is updated by machine learning. 22.The system of claim 21, further comprising: a sensor for detecting alocation of a user, wherein a projection direction of the directionalspeaker is controlled based on the location of the user.
 23. The systemof claim 21, wherein the audio converter is further configured togenerate an output for at least one nondirectional speaker and projectthe output of the at least one nondirectional speaker so that the outputis audible to all users in the room.
 24. The system of claim 21, furthercomprising: at least one nondirectional speaker enclosed in the speakerhousing.